16 research outputs found
Adaptive Proximal Gradient Method for Convex Optimization
In this paper, we explore two fundamental first-order algorithms in convex
optimization, namely, gradient descent (GD) and proximal gradient method
(ProxGD). Our focus is on making these algorithms entirely adaptive by
leveraging local curvature information of smooth functions. We propose adaptive
versions of GD and ProxGD that are based on observed gradient differences and,
thus, have no added computational costs. Moreover, we prove convergence of our
methods assuming only local Lipschitzness of the gradient. In addition, the
proposed versions allow for even larger stepsizes than those initially
suggested in [MM20]
Resolvent Splitting for Sums of Monotone Operators with Minimal Lifting
In this work, we study fixed point algorithms for finding a zero in the sum
of maximally monotone operators by using their resolvents. More
precisely, we consider the class of such algorithms where each resolvent is
evaluated only once per iteration. For any algorithm from this class, we show
that the underlying fixed point operator is necessarily defined on a -fold
Cartesian product space with . Further, we show that this bound is
unimprovable by providing a family of examples for which is attained.
This family includes the Douglas-Rachford algorithm as the special case when
. Applications of the new family of algorithms in distributed
decentralised optimisation and multi-block extensions of the alternation
direction method of multipliers (ADMM) are discussed.Comment: 23 page
Beyond the Golden Ratio for Variational Inequality Algorithms
We improve the understanding of the , which
solves monotone variational inequalities (VI) and convex-concave min-max
problems via the distinctive feature of adapting the step sizes to the local
Lipschitz constants. Adaptive step sizes not only eliminate the need to pick
hyperparameters, but they also remove the necessity of global Lipschitz
continuity and can increase from one iteration to the next.
We first establish the equivalence of this algorithm with popular VI methods
such as reflected gradient, Popov or optimistic gradient descent-ascent in the
unconstrained case with constant step sizes. We then move on to the constrained
setting and introduce a new analysis that allows to use larger step sizes, to
complete the bridge between the golden ratio algorithm and the existing
algorithms in the literature. Doing so, we actually eliminate the link between
the golden ratio and the algorithm. Moreover, we improve
the adaptive version of the algorithm, first by removing the maximum step size
hyperparameter (an artifact from the analysis) to improve the complexity bound,
and second by adjusting it to nonmonotone problems with weak Minty solutions,
with superior empirical performance
Over-the-Air Computation for Distributed Systems: Something Old and Something New
Facing the upcoming era of Internet-of-Things and connected intelligence,
efficient information processing, computation and communication design becomes
a key challenge in large-scale intelligent systems. Recently, Over-the-Air
(OtA) computation has been proposed for data aggregation and distributed
function computation over a large set of network nodes. Theoretical foundations
for this concept exist for a long time, but it was mainly investigated within
the context of wireless sensor networks. There are still many open questions
when applying OtA computation in different types of distributed systems where
modern wireless communication technology is applied. In this article, we
provide a comprehensive overview of the OtA computation principle and its
applications in distributed learning, control, and inference systems, for both
server-coordinated and fully decentralized architectures. Particularly, we
highlight the importance of the statistical heterogeneity of data and wireless
channels, the temporal evolution of model updates, and the choice of
performance metrics, for the communication design in OtA federated learning
(FL) systems. Several key challenges in privacy, security and robustness
aspects of OtA FL are also identified for further investigation.Comment: 7 pages, 3 figures, submitted for possible publicatio
Distributed Forward-Backward Methods for Ring Networks
In this work, we propose and analyse forward-backward-type algorithms for
finding a zero of the sum of finitely many monotone operators, which are not
based on reduction to a two operator inclusion in the product space. Each
iteration of the studied algorithms requires one resolvent evaluation per
set-valued operator, one forward evaluation per cocoercive operator, and two
forward evaluations per monotone operator. Unlike existing methods, the
structure of the proposed algorithms are suitable for distributed,
decentralised implementation in ring networks without needing global summation
to enforce consensus between nodes.Comment: 19 page